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Abstract: 

We  are  developing  a  synthetic  environment  to  be  used  in  combination  with  real 
environments  for  the  purpose  of  mine  counter  measures.  Two  examples  of  our  work 
involve  applications  to  land-based  minefields  and  an  application  to  submarine  defense.  In 
the  former  application,  imaging  is  done  with  six  spectral  bands.  It  has  been  shown 
empirically  that  a  sequence  of  images  taken  in  six  spectral  bands  when  viewed  sequentially 
will  allow  one  to  distinguish  between  real  mines,  partially  buried  real  mines,  decoys,  other 
metallic  objects,  and  other  manner  of  debris.  The  image  can  be  viewed  as  a  2-dimensional 
image  with  an  6-dimensional  vector  attached  to  each  pixel  location.  We  use  the  grand- 
tour  technique  (Wegman  and  Shen,  1993)  to  find  an  optimal  discrimant  between  real 
mines  and  other  objects.  We  then  use  a  head-mounted  display  (HMD)  which  is 
semitransparent  so  that  the  real-world  objects  can  be  seen  through  it.  After  processing  the 
scene,  the  suspected  sites  of  real  mines  are  superimposed  on  the  visual  field  so  that  the 
soldier  wearing  the  HMD  is  alerted  to  the  presence  of  mines. 

The  submarine  application  is  similar  in  that  multisensor  information  is  collected  for 
the  environment  surrounding  the  submarine.  In  effect,  we  have  a  low-dimensional  vector 
attached  to  each  voxel  in  the  3 -dimensional  image  surrounding  the  submarine.  Again  a 
grand  tour  is  performed  to  optimize  the  multisensor  data  for  detection  of  hostile  objects 
around  the  submarine.  In  this  application,  a  command  information  center  is  constructed 
with  3-dimensional  capable  projection  systems.  We  create  a  CAVE-like  environment  in 
which  each  wall  is  essentially  a  projection  screen  for  a  3-dimensional  image  of  the 
corresponding  underwater  environment  synthetically  reconstructed  based  on  our 
multisensor  data.  We  use  Crystal  Eyes  technology  and  Silicon  Graphics  Onyx  systems  in 
our  laboratory. 
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1.  The  Desktop  Metaphor,  Virtual  Reality,  and  Scientific  Visualization 

The  desktop  metaphor  is  a  common  conceptualization  of  a  human-computer 
interface  which  is  instantiated  in  many  current  computer  operating  systems  including  the 
MacOS,  Windows  3.1  and  Windows95,  and,  in  fact,  many  UNIX-based  operating  systems 
such  as  those  of  the  Silicon  Graphics  systems.  The  basic  idea  is  that  the  computer  screen 
is  in  effect  a  virtual  desktop,  that  windows  within  the  computer  screen  are  virtual  pieces  of 
paper,  and  that  by  using  a  mouse,  the  user  can  manipulate  this  virtual  desktop  in  a  way 
analogous  to  a  conventional  desktop.  The  computer  user,  of  course,  has  several  great 
advantages  over  a  conventional  paper-and-pencil  desktop  user  since  the  computer  is 
capable  of  a  much  richer  response.  That  is,  drawings  are  far  easier  to  make  using  a 
computer,  text  editing  is  far  easier,  dynamic  interaction  is  far  easier.  The  library  is  literally 
whatever  is  accessible  on  the  net.  Interestingly  enough,  constructing  and  writing 
mathematics  is  considerably  easier  with  a  conventional  desktop,  because  a  substantial 
extra  effort  is  required  for  the  writing  and  formatting  of  mathematical  arguments  in  a 
computer  system. 

To  a  large  extent  the  desktop  metaphor  is  satisfying  to  a  scientist  since  much  of  the 
analysis  work  that  a  scientist  does  is  desktop  oriented.  Thus  the  translation  from  a 
conventional  desktop  to  a  computer  environment  is  quite  satisfactory.  The  popularity  and 
prevalence  of  computing  within  the  scientific  community,  which  is  apparent  to  all,  is  a 
strong  indication  of  the  success  of  the  desktop  metaphor.  For  the  scientific  visualization 
community,  there  is  a  second  less  well  recognized  and  articulated  metaphor,  what  we  shall 
call  the  “explorer”  metaphor.  The  phrase,  exploratory  data  analysis,  now  a  phrase  in 
common  usage,  captures  the  essence  of  this  metaphor.  Projection  pursuit  and  high 
interaction  dynamic  graphics  are  additional  phrases  we  use  to  convey  an  exploratory  sense 
of  proactive  involvement  with  our  data.  Within  the  more  traditional  computer  science 
community,  the  phrases  data  mining  and  knowledge  discovery  again  attempt  to  capture 
the  exploration  of  data  and  databases.  It  is  our  assertion  that  these  phrases  implicitly 
represent  a  different  metaphor  from  that  of  the  desk-bound  analyst,  and,  in  some  sense, 
capture  this  more  “explorer”  metaphor.  It  is  interesting  to  note,  for  example,  in  a  recent 
paper,  Huber  (1994)  writes  of  exploratory  analysis  with  a  mountain  climbing  metaphor, 
the  Himalayan  metaphor.  Fortuitously,  the  mixing  of  these  metaphors  has  to  some  extent 
been  captured  in  several  successful  modern  windows-oriented  graphical  data  analysis 
systems,  notably  such  systems  as  Splus,  DataDesk  and  JMP. 

Our  thesis  is  that  so-called  virtual  reality  offers  a  better  technology  for  human- 
computer  interaction  as  it  relates  to  exploratory  analysis  than  does  the  desktop  metaphor. 
Virtual  reality  technology  not  only  offers  the  possibility  of  a  more  visceral  engagement  of 
the  data  world,  but  also  offers  a  much  more  compelling  visual  focus  than  does  a  standard 
computer  screen.  Regrettably,  virtual  reality,  like  artificial  intelligence,  expert  systems,  and 
neural  nets,  is  a  subject  of  much  hype  and  inflated  promises,  more  than  are  likely  to 
become  real  in  the  near  term.  For  the  reason,  we  have  routinely  chosen  phrases  like 
immersive  technology,  synthetic  environments,  and  augmented  reality  to  describe  our 


work.  We  briefly  describe  our  interpretation  of  the  explorer  metaphor  in  terms  of  virtual 
reality  technology. 

Wegman  and  Carr  (1993)  described  a  sea-exploration  metaphor  for  several  levels 
of  immersive  environments.  They  described  three  levels  of  immersion.  The  first  is  what 
we  think  of  as  exploring  the  undersea  environment  with  a  remotely  piloted  vehicle  (RPV). 
In  this  mode,  an  operator  typically  sits  with  a  joystick  controller  watching  a  television 
monitor,  and  maneuvers  the  RPV  with  the  joystick.  This  is  roughly  analogous  to  doing 
exploratory  data  analysis  with  a  mouse  on  a  computer  screen.  The  next  model  is  what  we 
have  called  the  aquarium  model.  This  is  like  studying  undersea  life  in  an  aquarium  with  a 
floor-to-ceiling,  wall-to-wall  glass  wall.  The  explorer  gets  a  3-dimensional  view  without 
being  seriously  encumbered  with  equipment.  In  our  instantiation,  we  use  a  three- 
dimensional,  stereoscopic  projection  system  which  gives  us  a  15-foot  diagonal  image  in  a 
stereoscopic,  full-color  display.  We'll  discuss  more  of  the  technical  details  later.  The  final 
model  is  what  we  have  referred  to  as  the  skin-diver  model.  The  analogy  is  strapping  on 
the  wet  suit,  the  oxygen  tanks,  and  the  face  mask  and  diving  to  explore  the  undersea 
environment.  The  computer  analogy  is  to  put  on  the  head-mounted  display,  the  tracker 
mechanism,  and  a  glove,  and  engage  the  computer-generated  virtual  world  directly.  As 
with  the  skin  diver,  there  are  some  equipments  that  must  be  used,  which,  truthfully,  are 
somewhat  cumbersome  and,  certainly  at  first,  awkward  to  use.  These  three  levels  of 
immersion  represent  three  levels  of  potential  exploratory  interaction  with  the  data  world. 

Obviously,  the  RPV  model  is  well  exploited.  We  have  exploited  the  aquarium 
model  and  have  generally  found  our  limited  explorations  to  be  quite  successful.  The 
projection  system  allows  for  the  high  resolution  desirable  in  a  graphical  analysis  and  the 
lack  of  physical  encumbrances  allows  a  focus  on  data  exploration.  The  skin  diver  model 
has  been  poorly  exploited  for  exploratory  analysis  to  date.  The  early  versions  of  head 
mounted  displays  were  either  high  resolution,  but  outrageously  expensive,  say  on  the 
order  of  US$100,000,  or  affordable,  on  the  order  of  US$6000  to  US$8000,  but  poor 
enough  in  resolution  so  as  to  be  unusable  for  statistical  data  graphics  purposes.  This 
situation  has  recently  changed  somewhat  with  the  introduction  of  the  Virtual  Research 
VR-4  head-mounted  display.  This  HMD  is  priced  at  about  US$8000  and  has  resolution 
comparable  to  S-VGA. 

Our  thesis  then  is  that  these  immersive  technologies  offer  a  data  exploration 
framework  that  has  the  potential  for  a  much  more  direct  interaction  with  the  data  world. 
The  major  thrust  of  our  work  is  to  combine  scientific  visualization,  exploratory  data 
analysis  and  virtual  reality  into  a  new  technology  for  exploring  data.  We  have  for  several 
years  experimented  with  3-D  visualization  and  have  many  aspects  of  this  tool  in  hand. 
Our  approach  has  been  to  treat  mathematical  constructs  as  if  they  were  real  entities,  i.e. 
treat  two-  and  three-dimensional  graphs  of  mathematical  functions  as  if  they  were  metallic 
surfaces,  and  visualize  them  in  three  dimensional  space  with  lighting  and  rendering  models 
similar  to  the  way  we  would  explore  a  CAD  design.  Figure  1  represents  a  two- 
dimensional  density  surface  derived  from  statistical  data  about  an  image  of  a  military  tank 
surrounded  by  foliage  and  other  highly  textured  picture  elements.  The  methodology  for 


identifying  the  target  involves  image  segmentation  techniques  based  on  fractal  geometry. 
Figure  2  represents  the  mass  distribution  of  supergalactic  clusters  with  the  three  variables 
being  declination,  right  ascension  and  red  shift.  Red  shift  is  a  surrogate  variable  for  radial 
distance.  Thus  Figure  2  quite  literally  represents  the  shape  of  space  at  the  most  highly 
aggregated  level. 

Data  analysis  has  seen  movement  of  two-dimensional  paper-based  graphics  in  the 
seventies  to  the  computer-based,  three-dimensional  color  graphics  of  the  eighties.  Three- 
dimensional  graphics  have  been  achieved  by  kinematic  displays  where  motion  gives  the 
depth  cues  and  by  stereoscopic  displays.  We  have  used  both  red-green  stereo  and 
polarized-light  stereo  to  good  effect.  However,  as  satisfying  as  these  displays  are  they 
clearly  represent  the  paradigm  of  a  scientific  investigator  looking  at  a  screen,  i.e.  the  RPV 
model.  What  we  are  doing  is  to  leave  this  scenario  and  immerse  the  scientific  data  analyst 
into  a  data  world.  Allowing  the  analyst  to  literally  move  about  in  the  data  world,  to  fly 
around  the  data,  to  look  at  different  portrayals  of  the  data,  to  make  a  turn  and,  literally, 
turn  into  another  dimension  seamlessly.  We  would  imagine  a  setting  where  an  analyst  can 
actually  grab  hold  of  a  data  point  in  this  virtual  world  and  see  the  effect  of  moving  it 
around.  We  would  imagine  a  setting  in  which,  as  the  analyst  approached  a  data  point  in 
this  virtual  data  world,  the  numerical  value  or  other  attributes  of  the  data  point  would 
come  into  focus.  We  are  creating  a  data  analysis  environment  where  the  scientist  could 
quite  literally  explore  the  data  much  the  same  way  that  an  oceanographer  with  scuba  gear 
or  a  submersible  could  explore  the  sea.  Figures  3  and  4  are  two  views  of  a  transparent 
rendered  head  based  on  MRI  data.  We  have  implemented  a  full  color,  transparent, 
stereoscopic  view  of  this  MRI  data  of  which  the  present  images  are  two-dimensional, 
gray-scale  renderings.  In  the  full  VR  view,  one  can  literally  see  internal  organs  and,  with 
fully  immersive  technology  such  as  a  HMD,  one  can  literally  walk  around  inside  of 
someone’s  head.  Color  anaglyph  stereo  images  of  these  and  other  visualization  images  are 
available  at  our  URL: 

http  ://www.  gal  axy .  gmu .  edu/images/gallery/research_arcade.  html . 

2.  Background  on  the  Mine  Warfare  Problem 

The  general  framework  of  our  research  has  been  oriented  to  the  visualization  of 
scientific  data.  From  this  work,  we  have  had  the  good  fortune  to  interact  with  researchers 
from  the  (U.S.)  Naval  Surface  Warfare  Center  in  Dahlgren,  Virginia,  with  the  NSWC 
Coastal  Systems  Center  in  Panama  City,  Florida,  and  with  the  Army  Research  Laboratory 
in  Aberdeen,  Maryland.  Suggestions  from  researchers  at  these  institutions  have  inspired 
us  to  apply  some  of  our  visualization  techniques  to  the  mine  warfare  problem.  This  paper 
is  a  description  of  the  problem  and  our  approach  to  addressing  the  problem. 

In  much  of  the  work  on  visualization  for  data  analysis,  spatial  extension  is 
considered  only  as  an  abstract  representation  of  a  data  variable,  but  not  as  any  real  entity. 
Although  we  have  been  interested  in  representation  and  visualization  of  multivariate  data, 
it  is  clear  then  there  are  many  circumstances  in  which  it  is  appropriate  to  measure  variables 
in  a  spatial  or  volumetric  setting.  Consider,  for  example,  the  setting  of  this  paper  in  which 


we  take  images  of  a  minefield  in  six  spectral  bands.  This  is  an  example  of  spatially 
extended  multivariate  data.  At  each  pixel  location,  we  have  six-dimensional  multivariate 
data  corresponding  to  the  intensity  levels  of  light  in  each  of  the  six  spectral  channels.  A 
minefield  may  be  two-dimensionally  extended  (spatially  extended)  in  the  case  of  land 
mines  or  in  the  case  of  underwater  naval  mines,  three-dimensionally  extended 
(volumetrically  extended).  An  alternate  civilian  application  would  be  to  the  visualization 
and  analysis  of  nuclear  magnetic  resonance  imagery  (MRI).  Here  we  are  concerned  with  a 
truly  volumetric  setting  in  which  multiple  sensors  may  give  us  partial  information  about 
tumor  site  targets.  Again  each  voxel  has  a  multidimensional  vector  attached  to  it  which 
may  have  many  missing  observations.  This  MRI  scenario  extends  in  an  obvious  way  to  an 
electronic  warfare  (EW)  electromagnetic  environment  as  well  as  to  other  potential  civilian 
applications. 

We  describe  most  specifically  the  land-mine  application.  This  application  is 
motivated  by  U.S.  Marine  Corps  interests  when  landing  on  a  possibly  mined  beachhead. 
Mines  may  be  laid  proud  (on  the  surface)  or  partially  buried.  Such  a  minefield  would 
generally  be  difficult  to  navigate  because  it  is  difficult  to  distinguish  among  the  various 
objects  with  the  unaided  eye.  However,  by  using  multiple  spectral  bands  including 
infrared  and  possibly  ultraviolet,  it  is  possible  distinguish  among  them.  For  example,  while 
a  real  mine  and  a  hollow  metallic  object  may  look  similar  to  the  unaided  eye  in  the  usual 
visible  light  range,  the  heat  absorption  characteristics  of  the  two  objects  would  be  different 
and  so  allow  one  to  distinguish  between  the  two  by  accounting  for  radiances  in  the 
infrared  band.  Similarly,  a  metallic  object  and  a  stone  object  generally  react  to  ultraviolet 
radiation  quite  differently  due  to  fluorescence  characteristics  and,  hence,  can  be 
distinguished. 

Experiments  at  the  Coastal  Systems  Center,  Panama  City,  Florida  involved  the 
construction  of  a  motion  picture  camera  which  interposed  a  different  spectral  filter,  up  to  a 
total  of  six,  for  each  frame  in  a  film  sequence.  Figures  5  through  10  represent  a  sequence 
of  six  images  of  the  same  minefield  from  a  film  taken  with  different  spectral  filters  ranging 
from  400  to  800  nm.  These  images  came  from  a  film  of  a  variety  of  objects  (mines, 
decoys,  stone  and  rocks,  man-made  bricks  and  concrete  objects,  wooden  objects  and  other 
debris  either  proud  or  slightly  buried)  on  soil  with  no  vegetation  although  other  images  are 
taken  in  a  variety  of  settings  including  in  shallow  water,  on  sand,  in  grass  and  low  shrubs. 
When  the  resulting  film  is  viewed,  it  is  quite  clear  that  the  rapid  sequencing  of  frames 
allows  one  to  clearly  distinguish  the  objects.  However,  because  of  the  strong  contrasts  of 
visual  properties  from  frame  to  frame  (i.e.  some  objects  bright  in  one  frame  and  very  dark 
in  the  next),  such  films  are  very  difficult  to  watch  causing  an  intense  fatigue  after  only  a 
few  minutes. 

Essentially,  the  Panama  City,  Florida  experiments  demonstrate  that  multispectral 
information  allows  a  human  viewer  to  distinguish  objects.  Because  of  the  persistence  of 
vision,  the  eye  effectively  integrates  the  multispectral  information  at  each  pixel  location 
forming  the  basis  for  the  ability  to  distinguish  objects.  Our  approach  is  to  recognize  the 
value  of  the  information  contained  in  the  multivariate  vector  at  each  pixel  location,  but 


rather  than  ask  the  human  eye-brain  system  to  process  the  information,  to  use 
mathematical/statistical  techniques  to  process  the  multivariate  vector  and  present  it  to  the 
viewer  in  a  more  palatable  form.  The  technique  we  will  describe  in  this  paper  is  called  the 
grand  tour  which  is  a  visualization  technique  for  multidimensional  data  that  we  have  used 
very  successfully  in  a  variety  of  scientific  visualization  problems. 

The  basic  idea  of  the  grand  tour  is  based  on  the  premise  that  to  thoroughly 
understand  an  object,  it  is  useful  to  look  at  it  from  all  points  of  view.  Briefly,  to 
understand  the  structure  of  a  cloud  of  multivariate  points  in  a  high  dimensional  space,  one 
can  simply  perform  a  time-dependent  general  rotation  of  the  multidimensional  coordinate 
axis  system  so  that  the  rotation  is  performed  in  a  continuous  (smooth)  manner  and  that  all 
possible  orientations  of  the  axes  are  eventually  achieved.  The  idea  was  original  formulated 
by  Asimov  (1985)  and  Buja  and  Asimov  (1985)  for  a  two-dimensional  grand  tour,  and 
later  by  Wegman  (1991)  for  the  general  d-dimensional  grand  tour  and  by  Wegman  and 
Shen  (1993)  for  a  one-dimensional  grand  tour.  The  d-dimensional  grand  tour  can  be 
visualized  using  the  parallel  coordinate  methodology  described  in  Wegman  (1990).  While 
the  one-dimensional  grand  tour  may  not  seem  so  useful,  for  imaging  applications  it 
emerges  as  a  quite  useful  technique,  since  the  result  of  a  one-dimensional  grand  tour  can 
be  rendered  in  grey-scale  at  each  pixel  location.  Thus  one  can  visually  search  for  an 
optimal  linear  combination  of  the  components  of  the  multidimensional  vector  which,  in  the 
case  of  our  application,  would  give  maximum  visual  distinction  of  real  mines  from  decoys 
and  debris.  Our  approach  concludes  by  highlighting  mine  locations  on  an  augmented 
reality  head-mounted  display  which  we  describe  in  a  bit  more  detail  in  Section  4.  In 
Section  3,  we  give  the  technical  details  of  one  of  our  grand  tour  techniques. 

3.  Visualization  Using  the  Grand  Tour 

The  goal  then  is  to  combine  the  information  in  the  multivariate  vector  attached  to 
each  pixel  or  voxel  in  such  a  way  as  to  make  the  objects  of  interest  visually  most  apparent. 
There  are  several  potential  approaches.  Some  simple  ones  are:  1)  to  use  different  channels 
(i.e.  vector  components)  for  each  eye  to  give  a  pseudo  stereo  effect  and  2)  to  display  each 
of  the  channels  in  rapid  sequence  so  as  to  create  a  scintillation  effect.  (We  have  already 
employed  this  technique  in  the  data  analysis  setting  to  good  effect.)  Both  of  these 
techniques,  while  potentially  effective,  are  unlikely  to  be  usable  on  a  sustained  basis 
because  of  the  eye  strain  and  fatigue  they  are  likely  to  cause.  A  more  sophisticated 
approach  is  the  one-dimensional  grand  tour  idea. 

To  explain  the  grand  tour  concept,  let  us  consider  a  point  in  d-dimensional  space. 
A  simple  device  for  doing  this  is  the  grand  tour.  The  grand  tour  concept  is  built  around 
the  idea  of  looking  at  an  object  from  all  points  of  view,  or  more  precisely,  in  all  frames  of 
reference.  The  original  work  of  Asimov  (1985)  constructed  all  possible  orientations  of 
two-dimensional  planes  and,  then,  viewed  the  data  as  being  projected  into  those  two- 
planes.  The  key  element  of  Asimov's  grand  tour  is  to  have  a  continuous,  space-filling  path 
through  the  manifold  of  two-planes.  Wegman  (1991)  considered  a  continuous,  space¬ 
filling  path  through  the  manifold  of  d-planes  in  order  to  construct  the  d-dimensional  grand 


tour.  However,  it  is  not  difficult  to  imagine  constructing  a  continuous,  space-filling  path 
through  the  set  of  straight  lines  in  d-space.  For  those  who  are  interested,  the  mathematics 
of  the  one-dimensional  grand  tour  are  developed  in  the  side-bar. 

We  use  the  one-dimensional  grand  tour  idea  at  each  pixel  or  voxel  location.  With 
this  device  we  can  examine  a  continuous  series  of  orthogonal  linear  combinations  of  the 
data  at  each  pixel.  We  use  the  same  grand  tour  at  each  pixel  location  so  that  as  we  move 
through  the  grand  tour,  we  discover  an  optimal  combination  of  the  components  of  the  data 
vector  which  maximally  discriminated  background  from  mines.  It  is  clear  that  we  need  a 
computationally  efficient  technique.  For  this  purpose  we  developed  pseudo-grand  tour 
plots.  The  mathematics  of  the  pseudo  grand  tour  are  developed  in  the  side-bar  labeled 
The  One-Dimensional  Pseudo  Grand  Tour. 

The  grand  tour  is  aimed  at  finding  a  useful  combination  of  the  multivariate  vector 
components.  In  the  classical  approach  to  the  grand  tour,  the  resulting  combination  of 
vector  components  is  viewed  through  a  dynamic  display.  Wegman  (1991)  and  Wegman 
and  Shen  (1993)  explain  how  this  can  be  done  in  the  general  d-dimensional  setting  and  the 
1 -dimensional  settings  respectively.  For  the  multispectral  image  setting,  we  have  in  mind 
the  following  technique.  In  the  above  formulation,  the  j  subscript  refers  to  the  indexing  of 
the  pixels.  Notice  that  the  grand  tour  vector  does  not  depend  on  the  pixel.  We  do  the 
same  grand  tour  for  every  pixel.  The  image  undergoing  the  grand  tour  is  displayed  in 
grey-scale  or  some  appropriate  color  scale.  We  allow  the  grand  tour  to  continue  until 
there  is  a  combination  of  terms  which  allows  the  targets  (mines)  to  be  visually  represented 
in  a  way  significantly  differently  from  the  background  clutter.  As  indicated  earlier,  the 
immersive  technology  allows  us  to  superimpose  the  grand-tour  derived  image  on  the  real 
scene  so  that  the  military  forces  could  navigate  through  mine  field  with  great  confidence. 

4.  The  Equipment  Base 

The  development  facilities  are  housed  in  the  virtual  reality  laboratory  of  our 
research  center,  the  Center  for  Computational  Statistics  at  George  Mason  University.  The 
facilities  have  been  under  active  development  for  just  over  30  months.  The  computer  suite 
used  directly  in  support  of  the  virtual  environments  facility  includes  two  Silicon  Graphics 
Onyx  RE  workstations,  a  Silicon  Graphics  Crimson  VGXT,  a  Silicon  Graphics  Indigo 
Elan,  and  a  Silicon  Graphics  Power  Series  4D/120/GTX.  The  total  hard  drive  capacity  for 
the  Silicon  Graphics  machines  exceeds  10.8  gigabytes.  The  virtual  reality  laboratory 
contains  five  head  mounted  display  systems,  three  by  Virtual  Research,  Inc.  and  two  by 
VI/O  technologies.  The  position  sensing  of  both  head  and  hand  is  accomplished  with 
Ascension  Technology's  Flock-of-Birds  Sensors  and  with  a  Polhemus  system.  The 
conversion  from  RGB  high  resolution  computer  display  to  NTSC  is  accomplished  with  a 
two  channel  encoder/decoder.  We  have  also  installed  a  Stereographies  high  resolution 
projection  system.  This  device  is  capable  of  1280x1024  resolution  and  projects  an 
approximately  15  foot  (five  meter)  diagonal  image.  It  is  driven  by  the  SGI  machines  and 
is  capable  of  stereoscopic  projection  using  Stereographies'  Crystal  Eyes  technology. 


We  supplement  the  Silicon  Graphics  computation  with  an  Intel  Paragon  A4 
concurrent  computer  with  61  nodes  capable  of  a  peak  speed  of  4.2  gigaflops.  These 
machines  are  tied  into  the  virtual  reality  labs  and  we  expect  to  use  these  machines  as 
compute  engines  for  our  virtual  environments  research.  In  addition  to  the  visualization 
capability  just  described,  we  will  have  a  five  channel  audio  subsystem  featuring  S-VHS 
and  laser  disk  capability.  The  virtual  reality  laboratory  also  contains  three  color  printers, 
one  a  Mitsubishi-Shinko  dye  sublimation  printer,  other  a  Hewlett-Packard  1200C/PS  ink 
jet  printer,  and  most  recently  a  QMS  color  laser  printer. 

While  the  development  environment  described  above  features  very  high  end 
equipment,  the  implementation  we  have  in  mind  ultimately  would  be  a  compact  computer- 
based  head-mounted  display  capable  of  what  we  like  to  term,  augmented  reality.  The  idea 
is  to  have  a  head-mounted  display  which  allows  the  user  the  capability  to  see  through  the 
display  system  to  the  outside  world.  The  concept  then  is  to  process  the  multispectral 
image  data  and  use  the  head-mounted  display  to  superimpose  warning  icons  on  the  real 
scene  to  warn  the  soldier  of  a  suspected  mine.  To  err  on  the  conservative  side  is,  of 
course,  desirable. 

The  head-mounted  display  developed  by  Virtual  I  O  Corporation  is  called  "i- 
glasses"  has  most  of  the  desirable  characteristics.  (Information  on  the  product  is  available 
on  URL  http://www.vio.com)  The  i-glasses  product  does  support  the  augmented  reality 
concept  by  allowing  the  user  to  view  the  external  world.  The  i-glasses  video  display  is  a 
lcd-based  stereoscopic  screen  of  180,000  pixels.  This  is  comparatively  low  resolution  and 
not  really  acceptable  for  a  final  military  product.  The  i-glasses  support  the  NTSC 
television  standard.  The  final  military  product  would  be  a  small  lightweight  integrated 
system  consisting  of  a  camera  system,  a  computer,  and  the  head-mounted  display.  The 
pseudo-grand  tour  algorithms  described  above  are  sufficiently  fast  that  for  the  180,000 
pixel  resolution,  a  Pentium-based  computer  is  adequate. 

5.  Current  Status 

The  general  work  on  scientific  visualization  using  virtual  reality  techniques  has 
been  well-developed  and  has  begun  to  emerge  in  the  technical  literature.  This  work  was 
funded  by  the  U.S.  Office  of  Naval  Research.  More  details  can  be  found  in  Wegman  and 
Carr  (1993)  or  at  URL  http://www.galaxy.gmu.edu.  That  work  has  drawn  the  attention 
of  several  of  the  naval  laboratories,  notably  our  colleagues  at  NSWC-DD,  Dahlgren, 
Virginia  and  NSWC-Coastal  Systems  Center,  Panama  City,  Florida. 


6.  ASW  Underwater  Mine  Scenario 

Closely  related  to  this  land  mine  is  the  immersive  techniques  for  underwater  mines. 
The  idea  is  similar  except  that  instead  of  a  two-dimensional  image  we  would  have  a  three- 
dimensional  voxel-based  view  of  our  environment.  The  other  significant  difference  is  that 


in  the  underwater  scenario,  we  would  not  have  a  multispectral  view,  but  a  multisensor 
view  which  would  tend  to  be  of  more  uneven  resolution  and  few  types  of  sensors  for  the 
underwater  setting  than  spectral  bands  in  the  land-based  setting.  Still,  the  fundamental 
idea  is  that  is  a  multivariate  data  vector  attached  to  each  voxel  location  which  can  be 
optimally  visualized  by  performing  a  one-dimensional  grand  tour.  Such  a  technique  would 
allow  the  visualization  of  targets  and  mines. 

A  rather  dramatic  implementation  could  be  conceived  for  this  setting.  We  have 
demonstrated  large-scale  stereoscopic  visualization  using  the  projection  techniques 
together  with  the  Crystal  Eyes  technology  mentioned  in  Section  3.  The  Crystal  Eyes 
technology  use  an  active  liquid  crystal  shutter  for  each  eye;  the  shutters  controlled  by  a 
computer  generated  infrared  signal.  The  stereoscopic  projection  system  is  capable  of  120 
frames  per  second  yielding  60  frames  per  second  for  each  eye.  Left  eye/right  eye 
perspectives  alternate  for  a  very  effective  stereoscopic  view.  One  can  imagine  a  command 
information  center  (CIC)  implementation,  say  on  a  submarine  where  each  of  the  six 
surfaces  of  a  cubical  room  has  a  stereoscopically  projected  image  using  the  Crystal  Eyes 
technology.  The  target  locations  based  on  the  processed  multisensor  data  would  be 
projected  on  the  walls,  ceiling,  and  floor  with  appropriate  stereo  depth  information  so  that 
the  commander  could  figuratively  see  through  the  walls  of  his  submarine  to  the  targets 
outside.  Of  course,  bottom  topography  could  be  included  either  from  sonar  or  elevation 
databases  or  both.  Texture  mapping  onto  the  elevation  database  could  given  a  pseudo- 
realistic  sense  of  the  position. 

We  have  at  this  point  demonstrated  pseudo-realistic  texture-mapping  onto  terrain 
databases  with  stereoscopic  visualization  as  described  above.  This  is  a  highly  effective 
technique.  We  have  also  used  the  Intel  Paragon  with  multiple  nodes  each  independently 
simulating  a  different  target.  This  work  is  still  in  progress. 

7.  Conclusions 

Our  work  has  demonstrated  a  combination  of  virtual  reality  in  the  form  of 
immersive  techniques  and  augmented  reality  coupled  with  statistical  visualization  tools  are 
extremely  useful  techniques  for  extended  the  soldiers  view  of  his  surroundings.  These 
techniques  have  arisen  out  of  an  interest  in  more  traditional  scientific  visualization  of  data, 
but  may  be  applied  effectively  to  military  scenarios.  Our  work  is  currently  in  process,  but 
shows  great  promise. 
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Legends  for  the  Figures 


Figure  1 .  Two-dimensional  density  surface  based  on  fractal  measurements  of  an  image  of 
a  camouflaged  military  tank  in  shrubbery  and  grass.  The  peaks  in  the  image  discriminate 
among  the  textures.  The  lighting  and  rendering  models  allow  for  visualizing  the  fine 
structure  on  the  density  surface. 

Figure  2.  Iso-contours  of  a  four-dimensional  density  surface  based  on  three  dimensional 
data.  The  data  are  the  mass  distribution  of  super-galactic  clusters  with  the  three  spatial 
variables  being  declination,  right  ascension  and  red  shift.  Red  shift  is  a  surrogate  variable 
for  radial  distance. 

Figure  3.  Profile  view  of  a  head  in  a  two-dimensional  rendering.  This  image  is  based  on 
MRI  data  which  was  reconstructed  to  a  full  three-dimensional  image,  then  rendered  as  a 
full  colored  stereoscopic  image  with  transparency.  The  internal  anatomical  structures  are 
visible  including  skin,  sub-dural  lining  brain,  facial  musculature,  tongue,  brain  stem  and 
even  the  cartilage  in  the  tip  of  the  nose.  MRI  responds  to  resonance  of  water  molecules, 
and,  hence,  primarily  to  soft  tissue  rather  than  bone  tissue. 

Figure  4.  Frontal  view  of  same  head.  Here  the  eyes  and  ear  structures  as  well  as  the  jaw 
musculature  are  clearly  visible. 

Figures  5  through  10.  Sequence  of  aerial  photos  of  a  minefield  with  other  debris.  Notice 
the  relative  brightness  of  pixels  within  the  minefield,  particularly  the  series  of  dots  in  the 
left  third  of  the  images  which  are  mines.  Their  brightness  relative  to  other  locations  is 
variable.  The  sequence  of  images  goes  from  an  infrared  filter  (Figure  5)  to  an  ultraviolet 
filter  (Figure  10).  These  images  have  been  contrast  enhanced  for  presentation.  The  native 
images  are  substantially  lower  in  contrast. 


Mathematics  of  the  Grand  Tour 

In  order  to  construct  the  one-dimensional  grand  tour,  let  us  consider  a  point, 
x  =  (*! . xd),  in  ^-dimensional  space.  This  point  x  is  the  multivariate  vector  attached 

to  each  pixel  or  voxel  location.  For  simplicity  of  exposition,  we  have  suppressed  the 
subscripts  identifying  the  pixel  or  voxel  location.  We  wish  to  visualize  points  of  this  form. 
Indeed,  we  wish  to  visualize  a  collection  of  such  points,  say,  of  size  n  of  the  form 
x  —  (x^,  ...,x^),  j  =  l,  2,  ....  n  where  n  is  the  total  number  of  pixels  or  voxels  under 

consideration.  We  wish  to  consider  >)-r  =  o  x  where  a  is  a  time-dependent  unit  vector, 

rsj  tr^j  j  r*J  t 

that  is,  we  take  a  =  (alt,a2 „ . . .  ,adt),  t  >  0  as  a  vector  for  which 

rsj  t 

d 

||  a  || 2  =  =  1  for  every  t. 

~  t  j=i  Jt 

Moreover,  it  must  be  the  case  that,  as  t  ranges  over  the  positive  reals,  a  ranges  over  all 

t 

possible  unit  vectors.  We  wish  to  have  a  be  a  continuous  function  of  t  and  exhaust  all 

r^J  t 

possible  unit  vectors,  that  is,  all  possible  orientations  of  the  unit  vector.  The  dot  product, 
y-t  =  a  x  yields  the  projection  of  the  x  data  vector  into  the  one-dimensional  manifold 

rsj  t  r-J  j  j 

spanned  by  the  unit  vector  a  .  According  to  the  traditional  view  of  a  grand  tour,  we  plot 

r^j  t 

yjt  on  a  fixed  one-dimensional  coordinate  system  for  time,  t  increasing.  This  is  the  movie 
view. 

The  grand  tour  in  two  dimensions  has  the  same  basic  set  up  as  in  the  one¬ 
dimensional  case  except  we  also  define  a  second  vector 

b  0 

ru  t 

where  b  is  a  unit  vector  such  that  a  b  =  0  for  every  t,  that  is,  a  and  b  are 

t  t  rsj  t  r*/  t  t 

orthonormal.  Next  let  y.=a  x  and  y\  =  b  x  .  Here  we  plot  y{  versus  y'  in  a  fixed 

ro  t  rsj  j  t  rsJ  j  3  3 

two-dimensional  coordinate  system  with  time,  /,  increasing.  Again,  this  is  the  movie  view. 


The  One-Dimensional  Pseudo  Grand  Tour 


Consider  now  the  J-dimensional  data  vector  x  =  (xlt  ...  ,  xd).  If  d  is  not  even, 
augment  x  by  one  additional  0,  so  that  x  =  (x,,  ...  ,  xd,  0).  We  may  assume  without  loss 
of  generality  that  d  is  even.  Define, 

a  '  =  (sinCw,  0,  cos(w,0,  •  •  •  ,  sin(wd/2t),  cos(a id/2t)). 

Then 

d/2 

II  a  '  II  2  =  (sin2(w/>  +  cos2(a>/))  =  d/2. 

j  =  1 

Hence,  we  rescale  by  y/2/d,  let  a  =  yjlid  a  '  which  implies  ||  a  ||  —  1.  For  technical 

t  rsj  /  rsj  t 

reasons,  w,  and  u.)  are  chosen  so  that  the  ratio  lo,/^  is  irrational  for  every  pair  (i,j)  and, 
moreover,  no  is  a  rational  multiple  of  any  other  u\/um,  i  /  k  and  j  /  m.  We  know 
that 

cos (u:/)  =  ±  y  1  -  sin2(a.y) 

so  the  scaled  vector,  a  will  not  exhaust  all  possible  orientations  and,  hence,  will  yield 

t 

only  a  pseudo-grand  tour. 

This  same  idea  can  be  extended  to  two  dimensions  in  order  to  construct  a  pseudo 
grand  tour  in  two-dimensions.  Let 

b  '  =  (cos -  sin(cj,/), . . .  ,  cos (ud/2t),  -  s\n(ujd/2t)). 

As  before 

d/2 

||  b  '  ||  2  =  £  (cos ?(ujQ  +  sin2(cj,/))  =  dfl. 

j=  1 

Hence  we  let 

b  =  yjlfdb  '  which  implies  ||  b  ||  =  1. 

t  r^>  t  r\j  / 

Moreover,  by  this  construction,  we  have 

d/2 

a  b  =  \  (sin(WjO  cos (wjt)  -  sin(u>/)  cos(u ;/))  =  0. 

rj  IfNj  1  j  =  1 

Thus  a  and  b  are  orthonormal  vectors  for  every  /  >  0.  Let  y\  —  a  x  and^y?  ==  b 

t  t  ft  tr^t  j  ft  r^j  t 

x  .  Plotting  y  versus  y\  for  every  j  as  t  ranges  over  positive  reals  yields  the  two- 

rsj  j  ft  ft 

dimensional  pseudo-grand  tour. 


slim*. 


&*  M  S. 


SF  298  MASTER  COPY 


KEEP  THIS  COPY  FOR  REPRODUCTION  PURPOSES 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  NO.  0704-0188 


Public  reporting  buraen  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources, 
garnering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comment  regarding  this  burden  estimates  or  any  other  aspect  of  this 
collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services.  Directorate  for  information  Operations  and  Reports.  1215  Jeherson 
Davis  Highway.  Suite  1204.  Arlington.  VA  22202-4302.  and  to  the  Office  of  Management  and  Budget.  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. 


3.  REPORT  TYPE  AND  DATES  COVERED 


AGENCY  USE  ONLY  (Leave  blank)  I  2.  REPORT  DATE 


April  1?96 


Technical 


4.  TITLE  AND  SUBTITLE 


Immersive  Methods  for  Mine  Warfare 


6.  AUTHOR (S) 


Edward  J.  Wegman,  Jeffrey  L.  Solka  and  Wendy  L.  Poston 


7.  PERFORMING  ORGANIZATION  NAMES(S)  AND  ADDRESS(ES) 

Center  for  Computational  Statistics 
George  Mason  University 
Fairfax,  VA  22030 


FUNDING  NUMBERS 


DAAH04-94-G-0267 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Research  Office 
P.O.  Box  12211 

Research  Triangle  Park,  NC  27709-221 1 


10.  SPONSORING  /  MONITORING 
AGENCY  REPORT  NUMBER 


ftp) 


11.  SUPPLEMENTARY  NOTES 


The  views, ^opinions  and/or  findings  contained  in  this  report  are  those  of  the  author(s)  and  should  not  be  construed  as 
"  '  ~  ment  of  the  Army  position,  policy  or  decision,  unless  so  designated  by  other  documentation. 


1 2a.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 


12  b.  DISTRIBUTION  CODE 


Approved  for  public  release;  distribution  unlimited. 


13.  ABSTRACT  (Maximum  200  words)  ^  ^  a  synthetic  environment  to  be  used  in  combination  with  real 

environments  for  the  purpose  of  mine  counter  measures.  Two  exanples  of  our  work  involve  applications  to  land- 
based  minefields  and  application  to  submarine  defense.  In  the  former  application,  imaging  is  dene  with  six 
spectral  bands.  It  has  been  shown  empirically  that  seqyenee  of  images  taken  in  six  spectral  bands  when  viewed 
sequentially  will  allow  one  to  distinguish  between  real  mines,  partially  buried  real  mines,  decoys,  other 
metallic  objects,  and  other  manner  of  debris.  The  image  can  be  viewed  as  a  2-dimensional  image  with  a  (ridimen- 
sional  vector  attached  to  each  pixel  location.  We  use  the  grand-tour  technique  (Wegman  and  Shen,  1993)  to  find 
an  optimal  discrimant  between  real  mines  and  other  objects.  We  then  use  a  head-mounted  display  (HID)  which  is 
semitransparent  so  that  the  real-world  objects  can  be  seen  through  it.  After  processing  the  scene,  the  suspected 
sites  of  real  mines  are  superimposed  on  the  visual  field  so  that  the  soldier  wearing  the  H©  is  alerted  to  the 
presence  of  mines. 

The  subnarine  application  is  similar  in  that  multisensor  information  is  collected  for  the  environment  surround¬ 
ing  the  subnarine.  In  effect,  we  have  a  low-dimensional  vector  attached  to  each  voxel  in  the  3r-dimensional 
imqgp  surrounding  the  subnarine.  Again,  a  grand  tour  is  performed  to  optimize  the  nultisensor  data  for  detection 
of  hostile  objects  around  the  submarine.  In  this  application,  a  command  information  center  is  constructed  with 
3-dimensional  capable  projection  systems. 

14.  SUBJECT  TERMS  I  15.  NUMBER  IF  PAGES 

Synthetic  environment,  virtual  reality,  mine  warfare,  augmented 

reality 


17  SECURITY  CLASSIFICATION  18.  SECURITY  CLASSIFICATION  19.  SECURITY  CLASSIFICATION  20.  LIMITATION  OF  ABSTRACT 
OR  REPORT  OF  THIS  PAGE  OF  ABSTRACT 

UNCLASSIFIED _ UNCLASSIFIED  |  UNCLASSIFIED  | _ UL _ 

Standard  Form  298  (R®v.  2-89) 

NSN  7540*01  -280-5500  Prescribed  by  ANSI  Std.  239-18 


16.  PRICE  CODE 


_GENERAL  INSTRUCTIONS  FOR  COMPLETING  SF  298 

thlt  thfJ3;t,,?«?r2^meCtat'on  P3Qe  (RDP)  is  used  in  announcing  and  cataloging  reports.  It  is  important 
in*orrr|at|?n  be  consistent  with  the  rest  of  the  report,  particularly  the  cover  and  title  paqe. 

opS^^mnin^nquI^lnw^'  of  the  f°rm  f°ll0W'  11  'S  important  t0  staV  within  the  lines  to  meet 


Block  1. 


Block  2.  Report  Date.  Full  publication  date 
including  day,  month,  and  year,  if  available  (e.g. 

1  Jan  88).  Must  cite  at  least  year. 

Block  3.  Type  of  Report  and  Dates  Covered. 
State  whether  report  is  interim,  final,  etc.  If 
applicable,  enter  inclusive  report  dates  (e.a 
10  Jun  87  -  30  Jun  88). 

Block  4.  Title  and  Subtitle.  A  title  is  taken  from 
the  part  of  the  report  that  provides  the  most 
meaningful  and  complete  information.  When  a 
report  is  prepared  in  more  than  one  volume, 
repeat  the  primary  title,  add  volume  number,  and 
include  subtitle  for  the  specific  volume.  On 
classified  documents  enter  the  title  classification 
in  parentheses. 

Block  5.  Funding  Numbers  To  include  contract 
and  grant  numbers;  may  include  program 
element  number(s),  project  number(s),  task 
number(s),  and  work  unit  number(s).  Use  the 
following  labels: 


C  -  Contract 
G  -  Grant 
PE  -  Program 
Element 


-  Project 

-  Task 

-  Work  Unit 
Accession  No. 


Block  6.  Authorfst.  Name(s)  of  person(s) 
responsible  for  writing  the  report,  performing 
the  research,  or  credited  with  the  content  of  the 
report.  If  editor  or  compiler,  this  should  follow 
the  name(s). 


Block  7. 


mm— Bgm 


Self-explanatory. 


Block  8.  Earforminq  Organization  Report 
Number,  Enter  the  unique  alphanumeric  report 
number(s)  assigned  by  the  organization 
performing  the  report. 


Block  9. 


Block  10. 


mtorin 


Self-explanatory. 

sorina/Monitoring  A 
(If  known) 


Block  11.  Supplementary  Notes.  Enter 
information  not  included  elsewhere  such  as; 
prepared  in  cooperation  with...;  Trans,  of...;  To  be 
published  in....  When  a  report  is  revised,  include 
a  statement  whether  the  new  report  supersedes 
or  supplements  the  older  report. 


Block  12a.  Distribution/Availabilitv  Statement. 
Denotes  public  availability  or  limitations.  Cite  any 
availability  to  the  public.  Enter  additional 
limitations  or  special  markings  in  all  capitals  (e.q. 
NORFORN,  REL,  ITAR). 

DOD  -  See  DoDD  4230.25,  “Distribution 
Statements  on  Technical 
Documents." 

DOE  -  See  authorities. 

NASA  -  See  Handbook  NHB  2200.2. 

NTIS  -  Leave  blank. 


Block  12b. 


DOD  -  Leave  blank 
DOE  -  Enter  DOE  distribution  categories 
from  the  Standard  Distribution  for 
Unclassified  Scientific  and  Technical 
Reports 

NASA  -  Leave  blank. 

NTIS  -  Leave  blank. 


Block  13.  Abstract.  Include  a  brief  (Maximum 
200  words)  factual  summary  of  the  most 
significant  information  contained  in  the  report. 

Block  14.  Subject  Terms.  Keywords  or  phrases 
identifying  major  subjects  in  the  report. 


Block  15.  Numbe 
number  of  pages. 

Block  16.  Price  C 
code  (NTIS  only). 


L.  Enter  the  total 


Enter  appropriate  price 


Block  17.-19.  Security  Classifications.  Self- 
explanatory.  Enter  U.S.  Security  Classification  in 
accordance  with  U.S.  Security  Regulations  (i.e., 
UNCLASSIFIED).  If  form  contains  classified 
information,  stamp  classification  on  the  top  and 
bottom  of  the  page. 


Block  20.  Limitation  of  Abstract  This  block  must 
be  completed  to  assign  a  limitation  to  the 
abstract.  Enter  either  UL  (unlimited)  or  SAR  (same 
as  report).  An  entry  in  this  block  is  necessary  if 
the  abstract  is  to  be  limited.  If  blank,  the  abstract 
is  assumed  to  be  unlimited. 


Standard  Form  298  Back  (Rev.  2-89) 


